Video karaoke system and method of use

ABSTRACT

In a method for aiding a user to customize a motion film selection, a film clip including a video track and a sound track comprising one or more actor voice tracks and a background track is selected, the sound track is modified to remove a selected actor voice track, the film clip with the modified sound track is displayed to the user, text representative of the selected actor voice track is displayed in synchrony with selected actor voice track while displaying the film clip to the user, a new voice track generated by the user is recorded for synchronized playback with the selected actor, and a new sound track including the modified sound track and the new voice track is saved.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/896,420 that was filed on Jun. 29, 2001 now U.S. Pat. No. 6,535,269and that claims the priority of provisional application serial No.60/215,513, filed Jun. 30, 2000.

BACKGROUND OF THE INVENTION

The present invention relates to video Karaoke systems, and moreparticularly to Karaoke systems that allow dubbing an actor's voice in avideo clip.

SUMMARY OF THE INVENTION

In one aspect, a method is provided for aiding a user to customize amotion film selection, comprising selecting a film clip including avideo track and a sound track comprising one or more actor voice tracksand a background track; modifying the sound track to remove a selectedactor voice track; displaying the film clip with the modified soundtrack to a user; displaying text representative of the selected actorvoice track in synchrony with selected actor voice track whiledisplaying the film clip to the user; recording a new voice trackgenerated by the user for synchronized playback with the selected actor;and saving a new sound track including the modified sound track and thenew voice track.

In a further aspect, the video track may be played together with the newsound track. The video track may be played together with the new soundtrack to synchronize the new voice track in the new sound track with theselected actor in the video track. A new film clip may be produced bysaving the video track together with the new soundtrack. The film clipmay be distributed for playback.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a system according to the invention; and

FIGS. 2-8 represent screens that may be displayed to a user of a videokaraoke system according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

With reference to FIG. 1, in one aspect the present invention provides asystem 10 for replacing the voice of an actor in the soundtrack of aselected movie clip with the voice of a user, and playing the movie clipwith the modified soundtrack. In one preferred embodiment shown in thefigure, the system 10 of the present invention comprises, generally, acomputer 20 with associated user command/selection input devices such asa keyboard 30 and/or a mouse 32, a touch screen, a remote control, aswell as audio input devices such as a microphone 36. The computer 20 isfurther connected to at least one video output device such as a typicalcomputer monitor 40, as well as audio output devices such as speakers[not shown]. The computer may also be connected to a typical TV set forplaying the modified movie clip. A user sound input device such as amicrophone 36 is also connected to the computer. The system 10additionally comprises at least one storage device such as a hard drive50 and/or a CD-ROM drive, CDR, CD/RW, DVD, and/or other removablestorage devices. The storage device may be physically connected to thecomputer 20 as is the case with typical hard drives, or may beaccessible through a communication link 60 (e.g. a modem, local areanetwork, wide area network, or storage area network) as is the casewhere the storage device is remotely located, such as a serveraccessible through the Internet, an intranet, via a dial-up connection,DSL, satellite, fiber optic lines, wireless networks, etc.

In addition to being connected to the storage device 50 to retrieve amovie clip for modification by the user, the computer 20 will also runthe software that enables the various functions of the system of thepresent invention. In a particular preferred embodiment, the software ofthe invention may perform the functions described below, which aredescribed in the order in which a user is likely to initiate them whileusing the system 10 of the invention.

The software may operate by displaying successive screens to the user onthe computer monitor 40, wherein each screen contains a number ofdifferent choices among which the user must select one. Thus, in onepreferred embodiment as illustrated by FIG. 2, an initial screen willpresent the user with the option of selecting a movie for a newrecording, playing a previously recorded performance, or reviewing helptopics. Upon choosing to make a new recording, the next screen as shownin FIG. 3 will display a list of types of movies clips available, suchas comedy, drama, action/adventure, science fiction, etc. Upon choosingone of these categories via the input devices 30 or 32, the followingscreen may present available movie clips belonging to the selectedcategory, as shown in FIG. 4. As seen in FIG. 5, after selecting aparticular clip the user may next be presented with the option ofpreviewing the clip or recording over one or more actors' voices.Previewing the clip will simply entail playing the clip on the monitor,or alternatively on the TV set, to allow the user to familiarize himselfwith the actors in the clip and the actors' lines (with or withoutcaptions), and to help the user decide which actor(s) to select. Thevideo clips may be stored in any preferred format, including but notlimited to the MPEG, AVI, Windows Media, QuickTime, or Real Videoformat. In an alternative embodiment, the software will also allow usersto provide their own video, such as from their own DVDs or camcorderclips, and to review the video clips to select a portion to revoice.

With continued reference to FIG. 5, another option may allow the user torecord over an actor's voice. A screen will first identify the actors inthe selected movie clip, and offer a list of those actors over which theuser may record his or her own voice. The user will then select one ormore actors, and will next proceed to record mode. If multiple audioinput devices are available, the system will allow the user to assignwhich actor's lines will be replaced by which input device, thusallowing multiple users to record the same scene simultaneously. Inrecord mode, the software will play the video image of the selectedmovie clip along with the background sounds, the music score, and thevoices of all the actors except the actor(s) selected by the user forvoice-over. As the selected actor delivers his or her lines in themovie, the actor's voice(s) are muted, optionally, as illustrated inFIG. 6, the actor's lines are also displayed as text onto the monitor40. Each sentence spoken by the selected actor preferably is preceded bycueing beeps to alert the user. As the selected actor delivers his orher lines, each word spoken by the selected actor may optionally also behighlighted across the screen as it is being spoken by the actor. Thisfeature helps the user synchronize his voice with the voice of theselected actor, and thus results in a more “natural” end product as thevoice of the user is better synchronized with the lips of the selectedactor. Additional caption notations such as emphasis/softness can beadded to enhance the recording.

As the selected actor's words are optionally displayed and highlightedacross the monitor, the microphone 36 is activated and records theuser's voice as the user speaks the selected actor's lines insynchronization with the selected actor. Of course, the user may chooseto speak different words than those of the selected actor, thereby ineffect editing the movie scene to the user's liking. By optionallydisplaying the selected actor's lines and highlighting the words as theyare delivered, the user is aided in synchronizing his new words to theselected actor's lips if he so desires, thus enhancing the illusion thatthe actor is actually delivering the user's lines.

Once the user has finished recording over the selected actor's voice,the system software offers the option of previewing the clip with themodified soundtrack, i.e. with the selected actor's voice replaced bythe user's voice, or the option of saving a new version of the clip withthe modified soundtrack. At this point the user may also choose toselect one of the other available actors and replace that actor's voice,or to re-record over the voice of the originally selected actor.

While recording over the voices of additional actors, the user maychoose to hear the original voices of all other actors, or may opt tohear the modified soundtrack with the previously selected actors' voicesdubbed over. Alternatively, the user may select to hear the originalvoice of some actors and the dubbed voices of other actors. In apreferred embodiment, the system software saves the voice of each actorin a separate, digitally encoded audio file (such as in a WAV or MP3format file), and thus allows the user complete flexibility in choosingthe desired audio track for each actor. Thus, when the user chooses tosave a modified soundtrack, the system software in essence saves anaudio file containing the user's voice for replacing an actor's originalvoice audio file during playback of the movie clip.

The user can repeat the recording steps for all available actors in theselected clip until the user is satisfied with the results, at whichtime the movie clip may be displayed full screen together with themodified soundtrack comprised of the individual sound files for eachactor, each actor dubbed over by the user, the background sounds, andthe music score. The modified movie clip may be displayed on thecomputer monitor 40 or an optional TV set 42 connected to the computer40 running the system software of the present invention.

In a preferred embodiment, the system software of the present inventionoffers additional features and options. Basic features such asmicrophone sensitivity, volume, brightness/contrast are provided.Noise-canceling features that record ambient noises and cancel them areused to improve audio quality. A desirable feature may include theability to adjust the properties of each audio file, including but notlimited to volume, pitch, tone, echo, and noise filtering, as shown inFIG. 7. A sound analysis plug-in system adhering to current standardswill allow users to add from a large collection of existing filters orcustom ones. In this manner, the user may adjust his recorded voice tomore closely match the voices of the other actors in a selected movieclip, and thus enhance the overall quality of the modified soundtrack byblending the modified voices with the original voices and the musicscore. Additional features may be provided to automatically adjust therecorded tracks with the original by “audio morphing” the recordedtracks to (variably) match the original. Static elements such as volume,amount of echo, can be automatically adjusted. Furthermore, timing canbe analyzed for pauses and individual vowels and then shifted, stretchedor compressed to match the original, further enhancing the realism ofthe user speaking the original lines. Similarly, spectral changes can beapplied or enhanced automatically to seamlessly immerse the user in thescene. The amount of “assistance” is user-selectable, or in a possiblegame scenario, reduced as the level of difficulty increases. Also, thedegree of assistance needed can be used to determine the score. As shownin FIG. 8, the user may choose to save any such modified, or adjusted,audio files as separate files in addition to the original replacementaudio file, and may thus choose to create multiple versions of a single‘take’. In this manner the user will be afforded the ability to comparethe various versions and thus select a preferred version. Comparing thevarious versions may be further aided by the use of a sound analyzer, asdescribed below.

Another desirable feature may be a sound analyzer that will display thewaveform and spectral analysis of a selected audio file, or optionallydisplay and contrast the waveforms of two or more selected audio files.In this manner, the user may compare his voice with the original voiceof the selected actor, and compare his timing, inflection, and volumeagainst the original actor's voice. This feature may also be useful foraspiring actors practicing their delivery, or alternatively for an userattempting to learn a foreign language by dubbing over a soundtrack inforeign language and using the system of the present invention to reviewand critique his progress. Additionally, the system may be used inspeech therapy, such as to teach the deaf or hearing impaired to speakby allowing them to dub over an actor's voice as they watch the actor'slips move and concurrently read the highlighted words, and subsequentlyto visually review their performance by comparing their voice with thatof the actor through the sound analyzer waveforms.

The sound analyzer may also be able to assign a numerical score or gradeto a user's recording based upon the degree to which the user was ableto match the original actor's voice. The score may be based upon amathematical analysis of the differences between the two waveforms interms of synchronization, volume, voice pitch, etc. The user optionallymay be allowed to select the parameters upon which the score is based,thereby allowing the user to tailor the scoring tool of the soundanalyzer to his particular desired use. Thus, the system of theinvention may be used as an entertainment tool that allows various usersto compete against one another in trying to dub over (or imitate?) anactor's voice. Alternatively, therapy or educational users of the systemmay assess their progress by keeping track of their scores.

Another option available to the user may be to save the movie cliptogether with any one or more of the modified voice tracks and theoriginal music score and background track as a standard video file, suchas in MPEG, AVI, Windows Media, QuickTime, or Real Video format. In thismanner, the user may distribute the modified movie clip to friends andothers via any available distribution medium, including either viae-mail or on a disc, tape, or cd. This feature thus opens many otherpossibilities for use of the present invention, as more fully discussedbelow.

Yet another option that may be offered by the system software of theinvention is editing the visual elements of the movie clip. Users canselect photographs to be superimposed/inserted into the video, thusinserting themselves into the scene. Another possibility is to segmentout the static background of a scene and replace it with anotherbackground or even video, thus completely altering the context of thevideo. In one embodiment, the computer 20 may additionally be connectedto a video camera [not shown] that will capture either still images ofthe user, or optionally capture a video file of the user's head as hespeaks the selected actor's lines. The software may embody functionsthat will replace the head of the selected actor with the videotapedimage of the user's head, thus in essence completely replacing theselected actor from the clip with the user. Optionally, a ‘canned’ orpre-recorded video source may be used to provide images to replace aselected actor, such a video tape, video disc, DVD, or a video computerfile in any desired format, such as MPEG, AVI, Windows Media, QuickTime,or Real Video. A more sophisticated setup can be used to scan the 3-Dstructure of a user's face or entire body, and using 3-D renderingtechniques, generate arbitrary movements of the user's virtual self andinsert or overlay them onto the scene. The actions of the virtualcharacters can be animated via several methods.Pre-recorded/pre-programmed motion sequences of the original actors canbe provided so their original actions can be mimicked exactly. However,the user can choose to animate their virtual self by specifying themotions manually, or in a sophisticated setting, use motion-capturetechnology to automatically convert the user's motions onto the screen.If desired, their motions can be judged by how closely they resemble theoriginal, in addition to scoring the revoicing.

As previously mentioned, the invention may be used for educationalpurposes, such as learning a foreign language. In this method of use ofthe invention, movie clips in the desired foreign language are madeavailable along with the appropriate text for each actor's lines. Theuser thus speaks along with a selected actor in the foreign languagewhile watching the actor's lips and reading along with the highlightedwords, thereby practicing not only his vocabulary but also the properinflection and timing. As discussed elsewhere in the specification, anaudio analyzer may be provided to display a visual indication of thematch between the user's voice and the original actor's voice, thusfurther aiding the user.

Similarly, the system software may be used to teach young children orthe learning impaired simple concepts such as, among others, objectrecognition. In one embodiment, a movie clip may be prepared wherein anactor points to various objects and concurrently pronounces eachobject's name. A child viewing the movie clip would repeat the name ofeach object while viewing the name of the object on the screen. In thismanner, the child would expand her vocabulary while learning to read, orrecognize letters, in an entertaining and amusing manner afforded by theability to hear her own voice played along with the movie clip.

As discussed above, in one embodiment of the invention each movie clipis comprised of a video file, individual audio files for each actor,background, and music, and multiple text files for each individualactor's audio file. In one method of preparing these files, a typicalvideo file such as a MPEG file of the desired movie clip is used toobtain the respective individual files listed above. Each individualactor's audio file is prepared by analyzing the actor's voice and thensubtracting that particular frequency range from the original soundtrackto obtain the particular actor's voice. When all actors' voices havebeen subtracted out and saved in individual files, the remainingsoundtrack comprises the background and music score. (Not this easy . .. )

In an alternative approach, the movie clip may be prepared for use withthe invention from the very beginning, by recording each actor's voiceindependently of the other actors, the background, and the music score.Professionally produced films and features are typically produced inthis manner, and thus one method of preparing movie clips for use withthe invention is to use the original studio video and audio recordingsto prepare the various individual audio tracks. Additionally, audiotracks on consumer video formats are encoded in multiple channels, suchas Dolby Surround, Dolby Digital, and DTS. In these formats the voicesare separately encoded in the center channel, and the rest of thechannels are the background and soundtrack. Therefore, the centerchannel can be decoded and broken down to individual actor's lines forrevoicing.

The text files associated with the individual audio files may beindividual video files of the movie clip with the text superimposed ontothe video image. In this embodiment, a video file with the correspondingtext superimposed upon the image is provided along with each individualactor's audio file, and the respective video file is played along withthe matching audio file whenever the user selects that particular actorfor over-dubbing. Alternative methods of displaying the text are withinthe ready purview of those skilled in the art, and are all includedwithin the scope of the invention as described herein. Other methods ofdynamically superimposing caption text can be built into the softwarepackage. It would render the text using the timing code and annotationsincluded in the text file.

The preferred embodiment of the invention entails the use ofprofessionally prepared movie clips, wherein the end user is providedwith a movie clip comprising the video file, the various individualaudio files, and the associated text files. In an alternativeembodiment, the system software of the invention may include functionsthat allow the user to select any video clip from any source and preparethe individual audio tracks and text files. The text may be encodedmanually by the user, or alternatively voice recognition software may beincluded in the system software to extract text information from audiofiles.

The preferred embodiment of the invention comprises software executableon a computer such as a personal computer or Macintosh running theWindows or Macintosh operating system, respectively. However, any devicecapable of executing the software code may be employed with theinvention, such as the Sony Playstation II and similar video gamedevices.

In another aspect of the invention, the system software will beavailable to users for purchase on various media, such as disks, cd's,or on-line via e-mail or the Internet. Similarly, movie clips comprisingthe necessary audio, video, and text files will also be available forpurchase through various media. Thus, in one embodiment, a serviceprovider may establish an Internet web site that users may log onto andelectronically purchase any of a variety of movie clips. Such clips maybe purchased by users for home entertainment purposes, or for use atpublic facilities such as bars, amusement parks and video arcades,charged on a per-use basis. To protect the intellectual properties ofcopyrighted material, security measures must be used when a video clipis distributed online. Specifically, watermark technology will be usedto encode information about the owner of the downloaded material. Todiscourage illegal distribution, the user's personal information, suchas name, address, and even credit card numbers, is strongly encryptedand then embedded in the video file, which can be recovered by thecopyright holder if necessary. If piracy is suspected, the originalbuyer of the video can be recovered and traced, thus acting as adeterrent to broadly trade files with strangers. This method will notprevent sharing between family and friends, which should actually beencouraged. However, possibly only degraded or preview versions of thevideo could be redistributed by the users.

In case of an accidental key loss, such as harddisk crashes, the usersimply has to provide the information used when purchasing the videos.However, if the information is lost, such as very old and long expiredcredit cards, the user will have the option of providing a new creditcard number. If the new information is valid, the customer will not becharged and a new key will be issued using the old information from theserver's database, so the video can be once again be unlocked. In thecase of falsely reported key loss, this scenario is detected during therenewal process, when the rightful owner renews with the originalinformation, which contradicts the new key. Therefore, the user whofalsely reported the lost key will have to pay for past usage to receivea renewal.

To protect the user's privacy, all online transactions will be donethrough secured channels, such as SSL (secured socket layer), andstrongly encrypted. To prevent possible hackers from recovering creditcard numbers, the credit card numbers are encrypted using one-wayciphers and thus non-recoverable. It is only necessary for the player touse the same cipher algorithm to encrypt and compare the encryptedcredit card numbers, not the original information.

To prevent key theft, or “identity theft,” user keys have to be renewedon a periodic basis with the server. This mechanism is for the users toupdate the information, such as address changes and credit card updates.During the renewal process, user information is verified, and a newencryption key is sent to the user. This key will be used to “unlock”the original video. Thus, in the event of a theft, the video/key pairwill only work until the next retrieval period.

In another method of distributing movie clips to users, promotionalclips may be freely distributed by movie studios to generate interest innewly released movies. Thus, in one embodiment, the service provider'sweb site may offer movie clips for sale as well as promotional movieclips free of charge. Promotional movie clips may not only be clips fromnewly released movies, but also television commercials featuringcelebrities, music videos, variety show clips, classic news segmentssuch as political speeches, etc. Depending upon the level of interest,some of these clips may be offered for sale. Promotional clipsdistributed by the service provider may be paid for by a fee charged tothe promoter, either in the form of a flat fee or a fee based upon thenumber of times a particular promotional clip is downloaded by usersaccessing the service provider's web site.

In an alternative method of use of the invention, movies, news segments,variety shows, and other video files may be produced for use with thesystem software of the invention and distributed in this manner to endusers. In a particular example, a television news segment may berecorded in English and distributed to local television stations in aformat compatible with the system software of the invention. A Spanishlanguage local station may then use the system software of the inventionto dub over the English language commentary with the equivalent Spanishtranslation, and then broadcast the news segment to its local Spanishspeaking viewing audience. Other similar post-production uses, includingfinal film soundtrack editing, are also within the scope of theinvention. In this manner, the invention would enable highly individualcustomization of video clips by end users for local, targeteddistribution.

In another alternative method of use, the system software may be used toprepare video clips for electronic distribution as announcements,invitations, greetings, etc. Electronic greetings have become verypopular, wherein an invitee receives an e-mail directing her to connectto a particular web site and view thereon the invitation or announcementthat somebody has mailed to her, potentially along with advertisementsand other promotional materials. Instead of displaying a static pictureor a known movie clip, the present invention would allow users of theelectronic greeting services to prepare a highly individualized videoclip wherein their favorite actors or singers deliver their particularmessage. In this embodiment, the system software may be partially orfully resident on the electronic greetings web site, and downloaded ondemand by the user. The web site may offer a variety of movie clips fromwhich the user may select one to edit on his home computer, followingwhich the edited video clip would be distributed from the web site tothe recipients listed by the user. The cost of this service would bepassed on to the advertisers displaying banners and other advertisingmaterial that is viewed by each recipient that signs onto the web siteto view her invitation or announcement.

Alternatively, the greetings provider may e-mail the entire modifiedvideo clip to each recipient. In such case the video clip may have to bemodified to require less memory and thus less transmission bandwidth,such as by decreasing the size of the video image or the color depth orboth, or through the use of an appropriate compression algorithm. Insuch an embodiment, the cost of the service may be charged to the userwho sends out the invitations, or to advertisers whose advertisingmessages may be automatically incorporated into the modified video clipor into the e-mail message that is sent to each recipient along with thevideo clip.

As mentioned above, the invention may be used for entertainment purposesin public venues such as bars or arcades. In one embodiment of such use,a booth may be provided containing the system 10 of the invention,wherein users may prepare custom video clips and e-mail them to variousrecipients for a fee.

Video compression technology has been steadily improving, with thecurrent state-of-the-art MPEG 4 reducing the file size two to ten foldwhen compared to the current MPEG 2 standard used in the DVD format andthe DSS satellite TV systems. Furthermore, bandwidth has been increasingquickly with cable modems, DSL, satellite, and optical networkingbecoming increasingly available. With the progression of reducing filesize and increasing bandwidth, online video distribution becomes moreviable with time. With available technologies today, short clips can bedownloaded rapidly. Other options can be to set up download stations atphysical locations, such as video rental stores, where removable storagecan be used to download and store custom selections of clips.

Other features and uses of the system and method of the invention mayinclude:

Distributing free versions of the player, downloadable through theInternet;

Enhanced versions for recording, plug-ins, works with existing DVDs;

Video games;

Sports newscasting;

Multi-user blind recording, simultaneous or sequential;

Imitating one or more actors in a film clip;

Imitating one or more singers in a music video;

Public competition/judging of best voice-overs;

Visual aids such as for lips shape/movement, tongue placement/movement;

Business presentations, such as annotation for commentary track forreview and search of video clip

Compression for distribution

Online distribution (server/user), including:

1. requesting clips,

2. previewing version,

3. purchasing selected clips,

4. encrypting the clips with a watermark including buyer info

5. player extraction of encrypted info from watermark and comparison tobuyer info on user's system (buyer's keys)—if the information matches,unlocking the film clips,

6. user sharing videos with friend—preview version is generated withwatermark intact, and

7. friend receiving preview version, which can be played without theneed for the original buyer's info friend can then purchase full-qualityvideo from server with their own info.

Having now described the invention in accordance with the requirementsof the patent statutes, those skilled in this art will understand how tomake changes and modifications in the present invention to meet theirspecific requirements or conditions. Such changes and modifications maybe made without departing from the scope and spirit of the invention asset forth in the following claims.

What is claimed is:
 1. A method for aiding a user to customize a motionfilm selection, comprising: selecting a film clip including a videotrack and a sound track comprising one or more actor voice tracks and abackground track; modifying the sound track to remove a selected actorvoice track; displaying the film clip with the modified sound track to auser; displaying text representative of the selected actor voice trackin synchrony with selected actor voice track while displaying the filmclip to the user; recording a new voice track generated by the user forsynchronized playback with the selected actor; and saving a new soundtrack including the modified sound track and the new voice track.
 2. Themethod of claim 1, further comprising: playing the video track togetherwith the new sound track.
 3. The method of claim 2, wherein playing thevideo track together with the new sound track comprises: playing thevideo track together with the new sound track to synchronize the newvoice track in the new sound track with the selected actor in the videotrack.
 4. The method of claim 1, further comprising: producing a newfilm clip by saving the video track together with the new soundtrack. 5.The method of claim 4, wherein saving the video track together with thenew soundtrack comprises: saving the video track and the new soundtrackwith the new voice track in the new sound track synchronized to theselected actor in the video track.
 6. The method of claim 4, furthercomprising: playing the new film clip.
 7. The method of claim 4, furthercomprising: distributing the new film clip for playback.